An Adaptive Approach to Collecting Multimodal Input

Multimodal dialogue systems allow users to input information in multiple modalities. These systems can handle simultaneous or sequential composite multimodal input. Different coordination schemes require such systems to capture, collect and integrate user input in different modalities, and then respond to a joint interpretation. We performed a study to understand the variability of input in multimodal dialogue systems and to evaluate methods to perform the collection of input information. An enhancement in the form of incorporation of a dynamic time window to a multimodal input fusion module was proposed in the study. We found that the enhanced module provides superior temporal characteristics and robustness when compared to previous methods.